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Abstract —Sampling above the Nyquist rate is at the heart of 
sigma-delta modulation, where the increase in sampling rate is 
translated to a reduction in the overall (mean-squared-error) 
reconstruction distortion. This is attained by using a feedback 
filter at the encoder, in conjunction with a low-pass filter at the 
decoder. The goal of this work is to characterize the optimal 
trade-off between the per-sample quantization rate and the re¬ 
sulting mean-squared-error distortion, under various restrictions 
on the feedback filter. To this end, we establish a duality relation 
between the performance of sigma-delta modulation, and that 
of differential pulse-code modulation when applied to (discrete¬ 
time) band-limited inputs. As the optimal trade-off for the 
latter scheme is fully understood, the full characterization for 
sigma-delta modulation, as well as the optimal feedback filters, 
immediately follow. 


I. Introduction 

Analog-to-digital (A/D) and digital-to-analog (D/A) con¬ 
verters are essential in modern electronics. In many cases, 
it is the quality of these converters that constitutes the main 
bottleneck in the system, and consequently, dictates its entire 
performance. On the other hand, as digital circuits are now 
considered relatively cheap to implement, the interface be¬ 
tween the analog and digital domains is often one of the most 
expensive components in the system. Developing A/D and D/A 
components that are on the one hand relatively simple, and 
on the other hand introduce little distortion, is therefore of 
interest. 

Often, the same A/D (or D/A) component is applied to 
a variety of signals with distinct characterizations. For this 
reason, it is desirable to design the data converter to be robust 
to the characteristics of the input signal. One assumption 
that cannot be avoided is the bandwidth of the signal to be 
converted, which dictates the minimal sampling rate, according 
to Nyquist’s theorem. Beyond bandwidth, however, one would 
like to assume as little as possible about the input signal. A 
reasonable model for the input signal is therefore a stochastic 
one, where the input signal is assumed to be a stationary 
Gaussian process with a given variance and an arbitrary 
unknown power spectral density (PSD) within the assumed 
bandwidth, and zero otherwise. In this paper, we adopt this 
compound model which is rich enough to include a wide 
variety of processes. The robustness requirement from the A/D 
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(or D/A) converter translates to requiring that it induces a small 
average distortion simultaneously for all processes within our 
compound model. 

Sigma-delta modulation is a widely used technique for A/D 
as well as D/A conversion. The main advantage offered by 
this type of modulation is the ability to trade-off the sampling 
rate and the number of bits per sample required for achieving 
a target mean-squared error (MSE) distortion. The input to 
the sigma-delta modulator is a signal sampled at L times 
the Nyquist rate (L > 1). This over-sampled signal is then 
quantized using an f?-bit quantizer. In much of the literature 
about sigma-delta modulation, no stochastic model is assumed 
for the input signal. However, when such a model is assumed, 
the benefit of over-sampling can be easily understood from 
basic rate-distortion theoretic principles: the (per-sample) rate 
required to achieve distortion D for the over-sampled signal 
is L times smaller than the rate required to achieve the same 
distortion for the signal obtained by sampling at the Nyquist 
rate. Thus, in principle, increasing the sampling rate should 
allow one to use quantizers with lower resolution, which is 
desirable in many applications. 

However, the rate-distortion theoretical property that guar¬ 
antees a constant product of the number of bits per sample 
needed to achieve distortion D, and the over-sampling ratio 
L, is only valid when a very long block of samples is vector- 
quantized. In A/D and D/A conversion, vector-quantization 
in high dimensions is a prohibitively complex operation, and 
quantization is invariably done via scalar uniform quantizers. 
Scalar quantizers alone cannot translate the increase of sam¬ 
pling rate to a significant reduction in the necessary resolution, 
but fortunately this problem can be circumvented with the aid 
of appropriate signal processing. 

In sigma-delta based converters, the quantization noise 
is shaped using a causal shaping filter embedded within a 
feedback loop, see Figure |T] The filter coefficients are chosen 
in a manner that ensures that most of the energy of the shaped 
quantization noise lies outside the frequency band occupied by 
the over-sampled signal. At the decoder, the quantized signal 
is low-pass filtered, cancelling out the high-frequencies of the 
quantization noise process without effecting the signal, such 
that the decoder’s output is composed of the original signal 
corrupted by a low-pass noise process. 

Another technique for compressing sources with memory, 
which explicitly models the source as a stochastic process, 
is differential pulse-code modulation (DPCM). In DPCM, a 
prediction filter is applied to the quantized signal. The output 
of this filter is then subtracted from the source and the result 
is fed to the quantizer, see Figure [2] At the decoder, the 
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quantized signal is simply passed through the inverse of the 
prediction filter. The well-known “DPCM error identity” 03 
states that the output of the decoder is equal to the source 
plus the quantization error, just like in simple non-predictive 
quantization. The benefit of using DPCM, however, is that the 
signal fed to the quantizer is the error in predicting the source 
from its quantized past, rather than the source itself. If the 
coefficients of the prediction filter are chosen appropriately, 
the variance of this error should be smaller than the variance 
of the original source, which translates to a reduction in the 
number of bits required from the quantizer for achieving a 
certain distortion. 

The performance of DPCM under the assumption of high- 
resolution quantization is well understood since as early as 
the mid 60’s HJ-EQ. Under this assumption, the prediction 
filter should be chosen as the optimal linear minimum mean- 
squared-error (MMSE) prediction filter of the source process 
from its past JT), and the effect of the filtered quantization 
noise can be neglected in the prediction process. While in most 
cases where DPCM is traditionally used, the high resolution 
assumption is well justified, it totally breaks down for the class 
of band-limited processes, which includes the input signals to 
sigma-delta modulators. Indeed, the prediction error of such 
a process from its infinite past has zero-variance, rendering 
the DPCM high-resolution rate-distortion formulas completely 
useless. 

A. Connection to Previous Work 

The connection between DPCM and sigma-delta modula¬ 
tion, as two instances of predictive coding, was known from 
the outset. Indeed, both paradigms emerged from two Bell- 
Labs patents authored by CC Cutler |4j, 0 in 1952 and 1954. 

In fact, by adding appropriate pre- and post-filters to the 
sigma-delta modulator, as depicted in Figure [3] the input to 
the quantizer, as well as the final reconstruction of the signal, 
become identical to those in the DPCM architecture 0 Section 
II], Q Chapter 3.2.4]. For this reason, it has become folklore 
that the two architectures are equivalent. When a sigma-delta 
modulator is used for compression of digital discrete-time 
signals, the pre-filtering can be performed digitally and the 
additional complexity of the architecture depicted in Figure [3 
w.r.t. that in depicted in Figured] may be acceptable. This is 
however not the case for data converters, as the input to the 
latter is analog and pre-filtering must be done in continuous¬ 
time, which is more challenging. The motivation for this work 
is understanding the performance limits of A/D and D/A 
conversion based on the sigma-delta architecture, and therefore 
pre-filtering is precluded. Thus, the architecture is confined to 
that depicted in Figure Q] 

Another important aspect of our interest in sigma-delta 
modulators as a mean of data-conversion rather then data- 
compression, is that it dictates that the assumptions one can 
make on the statistics of the input signal must be minimal. 
Consequently, we consider a compound class of sources that 
consists of all stationary Gaussian processes with variance <j 2 x 
whose PSD is limited to some predefined frequency band. In 


addition, since data converters often operate at very high rates, 
it makes sense to impose various constraints on the sigma- 
delta feedback filter C(Z), such as confining it to be a finite 
impulse response (FIR) filter with a limited number of taps. 
For a given desired MSE distortion level, our goal is to find the 
constrained sigma-delta feedback filter C(Z) that minimizes 
the quantization rate w.r.t. all sources in the compound class, 
and to characterize the attained rate. This goal is different than 
the one pursued in (8), where the optimal unconstrained filters 
w.r.t. a known PSD were found. 

The problem of finding the optimal A'-tap FIR sigma- 
delta feedback filter C(Z) for a compound family of sources 
similar to ours, was considered in 0. The optimal filter was 
claimed in 0 to be the Nth order MMSE prediction filter 
C{Z) = (1 — Z~ 2 ) N of a bandpass stationary process from 
its past, and for a fixed target MSE distortion the required 
quantization rate was found to decrease linearly with N. Such 
as statement is obviously inaccurate, as it violates Shannon’s 
rate-distortion theorem. The major drawback of 0 is that 
it (implicitly) makes the high-resolution assumption that the 
variance of the quantizer’s input is solely dictated by the target 
signal {X n }, whereas the contribution of the quantization 
noise to this variance can be neglected. As discussed above, 
for over-sampled processes this assumption may not be valid 
even when the quantizer’s resolution is very high. In particular, 
using the filter C(Z) = (1 — Z _1 ) N from 0 , the energy of the 
quantization noise within the frequency band occupied by the 
signal indeed decreases exponentially with N. However, the 
noise’s energy outside this band increases rapidly with N, and 
for any quantization resolution it will become much greater 
than a x for N large enough, making the high-resolution 
assumption inapplicable. In this case, the dynamic range of 
the quantizer will be exceeded and overload errors would 
frequently occur. 

It therefore follows that in the analysis of sigma-delta 
modulators one should not make high-resolution assumptions, 
but rather must take into account the effect of the filtered 
quantization noise on the variance of the quantizer’s input. 
Fortunately, in the analysis of DPCM modulators the high- 
resolution assumption has been overcome in 0. It was shown 
that for any distortion level and any stationary Gaussian 
source, the DPCM architecture induces a rate-distortion op¬ 
timal test channel, provided that the prediction filter is chosen 
as the optimal filter for predicting the source from its quantized 
past, and in addition water-filling pre- and post-filters are 
applied. The analysis of 0, which takes into account the 
effect of the quantization noise, can therefore be used to obtain 
the optimal feedback filter and its corresponding performance 
for a DPCM system applied to an over-sampled stationary 
Gaussian source. In this paper, we leverage the results from 0 
to the analysis of sigma-delta modulators, by establishing an 
appropriate duality between the two architectures. 

B. Contributions 

Let S be the compound class of all discrete-time stationary 
Gaussian sources with variance o x and PSD that is zero for 
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all uj (t [— 7r/L, tt/L\, L > 1. Note that this class corresponds 
to uniformly sampling a compound class of continuous-time 
stationary Gaussian processes with variance a\ and PSD that 
is zero for all |/| > / max , at a sampling rate of 2L/ max sam¬ 
ples/per second. Let {X° PCM } be a discrete-time stationary 
Gaussian process with PSD 


S£ PCM (w) 


La\ for |w| < 7 t/L 
0 for 7 t/L < |w| < 7r 


( 1 ) 


and note that {X° PCM } £ S. 

Our main result, derived in Section [ill is that for any process 
{X^ A } from the compound class S , the test channel induced 
by the sigma-delta modulator (Figure [TJ achieves exactly the 
same rate-distortion function as that of the DPCM test channel 
(Figure [2} with input {X° PCM }. More specifically, for such 
processes, for any choice of <7 q PCM and prediction filter C(Z) 
in the test channel of Figure [2] the same choice of C(Z) 
together with the choice 
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J2 ^DPCM /o\ 

a TA ~ -77- (A) 

l ■ ^ I-J/lI 1 - c ( u} )\ 2(Lj 

in Figure |7] yields the same compression rate and the same 
distortion. 

While this result is simple to derive, it has a very pleasing 
consequence: the problem of optimizing the filter C(Z) in 
sigma-delta modulation w.r.t. any signal in S, under any set of 
constraints, can be cast as an equivalent problem of optimizing 
the DPCM prediction filter w.r.t. input {X^ PCM } under the 
same set of constraints. Furthermore, in Section III-A1 we 
formalize a similar duality between DPCM and sigma-delta 
modulation for a frequency-weighted-mean-squared-error dis¬ 
tortion measure. In this case S'y PCM (^) is replaced with a PSD 
that depends on the distortion’s weight function. 

In principle, recasting the sigma-delta optimization problem 
as an MMSE prediction problem may be derived directly 
from the formulas characterizing its performance, as given 
in Proposition Q] Nevertheless, establishing the equivalence 
between sigma-delta modulation and DPCM, in the specific 
form described above, is insightful as it allows to borrow 
known results from the literature about the latter. 

Having recast the filter optimization problem for sigma-delta 
as that of optimal linear prediction, we can readily obtain the 
solution under constraints for which an explicit solution was 
lacking in the literature, or was cumbersome to derive. 

One may question the relevance of the test channel of 
Figure Q] and its information-theoretic analysis to the practical, 
resource limited, problem of A/D and D/A conversion via 
sigma-delta modulators. To that end, in Section [HI] we replace 
the AWGN channel from FigureQ]with a simple scalar uniform 
(dithered) quantizer of finite support, which is suitable for 
implementation within A/D and D/A converters. As long as 
overload does not occur, the effect of applying the scalar 
quantizer is equivalent to that of an additive noise channel. We 
show that the rate-distortion trade-off derived for sigma-delta 
modulation in Section [IT] remains valid with high probability, 
with a constant additive excess-rate penalty for using scalar 


quantization. The purpose of this excess-rate is to ensure 
that an overload event, which jeopardizes the stability of the 
system, occurs with low probability. The stochastic model we 
assume for the input process allows us to tackle the issue of 
stability in a systematic and rigourous manner, and the trade¬ 
off between the excess-rate and the overload probability is 
analytically determined. 

Clearly, a sigma-delta modulator can only perform well 
if overload errors are rather rare. Our stability analysis in 
Section m is based on avoiding overload events w.h.p., and 
does not aim to consider the effect of such events on the 
distortion once they occur. In general, the overload probability 
of the scheme described in Section [Hi] decreases double 
exponentially with the excess-rate of the quantizer w.r.t. the 
mutual information. Thus, taking an excess rate of 1 — 2 
bits will usually yield a sufficiently low overload probability. 
However, sigma-delta quantizers are often employed with a 
one-bit quantizer. In this case, the overload error probability 
cannot be very low. Consequently, the designer would need to 
guarantee that the effect of overload errors is local in time, and 
does not drive the system out of stability. There are various 
restrictions one can place on C(Z) in pursuit of the latter goal. 
The issue of maintaining stability when overload errors are 
unavoidable is outside the scope of this paper. Nevertheless, 
we stress that our main result is of great relevance to this 
setting, as it shows that the filter C(Z ) should be chosen as 
the optimal MMSE prediction filter of {X° PCM } from its noisy 
past under the stability ensuring restrictions. 

II. Main Result 

We begin by introducing some basic notation that will be 
used in the sequel. For a discrete signal {c n }, the Z-transform 
is defined as 

OO 

C(Z)± J2 c " Z ~ n ’ 

n =—oo 

and the Fourier transform as 

OO 

C(u)±C(Z) | z=e#w = ^ c n e-i“ n . 

n =—oo 

For a discrete (real) stationary process {X n } with zero-mean 
and autocorrelation function Rx[k] = E(X n+ kX n ) we define 
the power-spectral density (PSD) as the Fourier transform of 
the autocorrelation function 

OO 

Sx(u) = J2 Rx{k\e-i“ k . 

k=— oo 

The PSD of a continuous stationary process is defined in an 
analogous manner. 

Assume X EA (f) is a continuous stationary band-limited 
Gaussian process with zero mean and variance cr 2 x , whose PSD 
is zero for all frequencies |/| > /max, but otherwise unknown. 
The Nyquist sampling rate for this process is 2/ max samples 
per second. Since our focus here is on quantization of over¬ 
sampled signals, we assume that A' sa (t ) is sampled uniformly 
with rate of 2L/ max samples per second for some L > 1. 
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Fig. 1. The test channel corresponding to the sigma-delta modulation architecture, with the sigma-delta quantizer replaced by an AWGN channel. The input 
is assumed to be over-sampled at L times the Nyquist rate. 
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Fig. 2. The test channel corresponding to the DPCM architecture, with the DPCM quantizer replaced by an AWGN channel. The input is assumed to be 
over-sampled at L times the Nyquist rate. 
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Fig. 3. A test channel corresponding to the sigma-delta modulation with pre-filter 1 — C(Z) and post filter x—C(Z) ■ ThH test-channel is equivalent to that 
from Figure [2] However, the pre-filter makes this architecture unattractive for data converters. 


The obtained sampled process {X^ A } is therefore a discrete 
stationary Gaussian process with zero mean and variance a 2 x 
whose PSD is zero for all w ^ [— n/L, n/L], but otherwise 
unknown. Our goal is to characterize the rate-distortion trade¬ 
off obtained by a sigma-delta modulator, modeled as the test 
channel from Figure Q] whose input is {A'^ a }. To that end, 
we establish an equivalence between the performance obtained 
by this test channel for any stationary band-limited Gaussian 
process with variance cr\ and the performance obtained by the 
test channel from Figure [2] which models a DPCM compres¬ 
sion system, for a stationary flat band-limited Gaussian process 
with variance The performance of the latter is now well 
understood 0, and, as we shall show, can be translated to 
a simple characterization of the performance of sigma-delta 
modulation. 

First, we recall the derivation of the distortions attained 
by the test channels from Figure |T] and Figure [2] and 
the scalar mutual information I(U^ A ;U^ A + N^ A ) and 
/([/“; /7 ° pcm + N° FCM ) between the input and output of 


the additive white Gaussian noise (AWGN) channels embed¬ 
ded within the two test channels. 

The test channels in Figure Q] and Figure U do not imme¬ 
diately induce an output distribution from which a random 
quantization codebook with rate I(U n ;U n + N n ) and MSE 
distortion D can be drawn. The reason for this is the sequential 
nature of the compression, which seems to conflict with the 
need of using high-dimensional quantizers, as required for 
attaining a quantization error distributed as N n with compres¬ 
sion rate I(U n ; U n + N n ). Fortunately, this difficulty, which is 
also present in decision-feedback equalization for intersymbol 
interference channels, can be overcome with the help of 
an interleaver ED-HD- Thus, the scalar mutual information 
I ( U n ; U n + N n ) can indeed be interpreted as the compression 
rate needed to achieve the distortion attained by the test 
channels in Figure Q] and Figure [2] We elaborate further about 
this in subsection III-B I Moreover, in Section [III] we show that 
I(U n ', U n + N n ) is closely related to the required quantization 
rate in a sigma-delta modulator that applies a uniform scalar 
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quantizer of finite support. 

We begin with the test channel in Figure Q] that corresponds 
to sigma-delta modulation, with the sigma-delta quantizer 
replaced by an AWGN channel with zero mean and variance 
The filter C(Z) is assumed to be strictly causal. 

Proposition 1: For any Gaussian stationary process {X^ A } 
with variance whose PSD is zero for all u \—Tt/L,it/L\, 
the test channel from Figure |T] achieves MSE distortion 

D = 4 a ' TT [ |1 ~ C(iv)\ 2 duj, 

J-tv/L 

and its scalar mutual information satisfied 


I(U* A -,U* A + N* A ) 

= - log 11 + 


2n 


\C(oj)\ 2 du> 



Proof: From Figure Q] we have that 

/■/"SA vSA atSA 

— Cn * , 


(3) 


the optimal MMSE infinite length prediction filter of X° PCM 
from all past samples of the process {X° PCM + iV° PCM }. The 
following straightforward proposition characterizes the rate 
and distortion for any choice of the causal filter C(Z) and 
any value of 4 pcm- 

Proposition 2: For a Gaussian stationary process {X PPCM } 
with variance <j\ and PSD 


-rDPCM 

5 .y 


M = 


La\ for |w| < it/L 
^0 for 7 x/L < |w| < 7 t ’ 

the test channel from Figure [2] achieves MSE distortion 


(7) 


D = 


2 

^DPCM 


and its scalar mutual information satisfies 


DPCM. rrDPCM 




■N, 


DPCM 


) = o lo s 1 + 


2tt 


Lo\ 


t/L 


\C(uj)\ 2 du 


CT DPCM 2 7 T J-n/L 


|1 — C(u})\ 2 dui 


and therefore 


U: 


EA 


■N™=Xf* + (6 n -c n )*N™ 


where S n is the discrete identity filter. Using the fact that 
{ X ^ A } is a low-pass process, passing it through the filter 
H(uj) has no effect, and hence 

X^ = h n *(U^+N^) 

= X* A + h n * (5 n - c n ) * N* A . 


The MSE distortion attained by the test channel from Figure |T| 
is therefore 


D = E(X^ a - X^ A ) 2 = 4a ~ f' L |1 - C(w)\ 2 du,. 

The scalar mutual information between the “quantizer’s” input 
Un A and output U^ A + Nf A is given by 


I(U* A -,U* A + N* A ) 


= h(U* A 



+ N* A )-h(N* A ) 

i + ^p!V 


(4) 

(5) 


where ©, as well as ©, follow from the statistical indepen¬ 
dence of N^ a and U^ A . Using ©. the variance of U'f A is 


^(Un A ) 2 — 4 + ct EA^: j |C(w)| 2 dio. (6) 

Substituting © into © establishes the second part of the 
proposition. ■ 

Next, we analyze the test channel in Figure [2] that cor¬ 
responds to DPCM compression with the DPCM quantizer 
replaced by an AWGN channel with zero mean and variance 
4pew ■ As in the test channel of Figure |T] the filter C\Z) 
is strictly causal. The distortion corresponding to this test 
channel, as well as /(U° PCM ; U° PCM + X° PCM ), were already 
found in © Theorem 1] for the special case where C(Z) is 


1 All logarithms in this paper are taken to base 2. 


Proof: From Figure [2j we have that 

rrDPCM vDPCM „ rrDPCM 
U n = X n - C n * V n 

rrDPCM rrDPCM , a rDPCM , , rrDPCM 

V n =U n +N n +C n * V„ 

Substituting © in © yields 

rrDPCM _ xrDPCM I a^DPCM 
* n n 


( 8 ) 

(9) 

GO) 


Using the fact that {X° PCM } is a low-pass process, as before, 
we obtain 


X, 


DPCM 


= h n * (X™ + N° PCM ) 

= x n DPCM + K * n» pcm . 


(ID 


Since {X° PCM } is AWGN with variance 4pcm> the variance 
of the filtered process h n * N® PCM is 4 pcm 4- Thus, 

2 

jj _ g( yDPCM _ j^DPCM-j2 _ °~DPCM 

L 

As in the analysis of the test channel from Figure Q] the scalar 
mutual information between f/° PCM and [7° PCM + X° PCM is 
given by 

+ «r+ = i log (1 + ™rn 

1 \ ^DPCM / 

( 12 ) 

Now, substituting (ITOt in © gives 

C/DPCM = (Sn _ ^ * X DPCM _ ^ * ^DPCM^ 


and the variance of U n is therefore 

HU™™) 2 = ^ - C(u)\ 2 dut 

+ 5'a PCM 4)|C(w)| 2 ^ 

( 13 ) 
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Substituting ( IT3l ) into (1 1 2b establishes the second part of the 
proposition. ■ 

Remark 1: In propositions Q] and [2] we derived the scalar 
mutual information between the input and output of the 
AWGN test channels embedded in Figures Q] and [2] respec¬ 
tively. As will become clear in Section |III1 the scalar mutual 
information is closely related to the required quantization rate 
when a scalar memoryless quantizer is used within the sigma- 
delta or DPCM modulator. In Km. the directed informa¬ 
tion was shown to be related to the required quantization rate 
when the quantizer is followed by an entropy coder. Here, 
we do not consider applying entropy coding to the quantizer’s 
output as we require that the designed modulator be robust 
to the statistics of the input process, whereas entropy coding 
is very sensitive to the process statistics. Moreover, if the 
design of an A/D (or D/A) is considered, the appropriate merit 
for the modulator’s complexity is the number of quantization 
levels within the scalar quantizer, which are not reduced by 
incorporating an entropy coder. 

Our main result now follows immediately from Proposi¬ 
tions Q] and U 

Theorem 1: Let {X^ A } be any Gaussian stationary process 
with variance r t\ whose PSD is zero for all w [—7 t/L, t t/L\, 
let {X“} be a flat low-pass Gaussian stationary process 
with PSD as in (0, and let C(Z) be a strictly causal filter. 
The test channel from Figure Q] with 


SA i-c(«)iw 

and the test channel from Figure |2] with 
°dpcm = L ■ D, 


both achieve MSE distortion D and their scalar mutual infor¬ 
mation satisfy 


m 


SA - t/ n SA + 7V„ sa ) = /(t/° PCM ; t/° PCM + iV° PCM ) 


= - log 1 + 


27T 


+ 


1 


D 2t r 




\C(ut)\ 2 duj 

r/L 


— tt/L 


\l~C(u))\ 2 dco . 


This theorem indicates that for any stationary band-limited 
Gaussian process with variance <j 2 x , the sigma-delta test chan¬ 
nel from Figure |T] achieves exactly the same rate-distortion 
trade-off as that of the DPCM test channel from Figure [2] 
with a stationary flat band-limited Gaussian input with the 
same variance, provided that the AWGN variances are scaled 
according to ( 0 . Thus, Theorem [j] provides a unified frame¬ 
work for analyzing the performance of sigma-delta modulation 
and DPCM. A great advantage offered by such a unified 
framework, is that any result known for DPCM can be trans¬ 
lated to a corresponding result for sigma-delta modulation, and 
vice versa. Theorems [2] and Corollary Q] below constitute two 
important examples of such results. 


Theorem 2: Let {A^ a } be a Gaussian stationary process 
with variance a\ whose PSD is zero for all u> ^ [— 7t/L, n/L\ 
and let C be a family of strictly causal filters. Define the 
“virtual” process { S n } as a Gaussian stationary process with 
PSD as in ©, and the “virtual” process { W n } as a Gaussian 
i.i.d. random process statistically independent of {SV,,} with 
variance L ■ D, D > 0. Let 

o*n = min E (S n - c n * (S n + W n )f 
c(z)ec 

C* D (Z) = argminE (S n - c„ * (S n + W n )) 2 . 
c{z)ec 

If the filter C(Z) in the sigma-delta test channel from FigureH] 
belongs to C and the MSE distortion attained by this test 
channel is D, then 

I(t/ n SA ; C/„ SA + N* a ) >1 log (l + , (14) 

with equality if C(Z) = C£,(Z). 

Theorem [2] states that for a target distortion D, the sigma- 
delta filter which minimizes the required compression rate 
is the optimal linear time-invariant MMSE estimator, within 
the class of constraints C, for S n from the past of the noisy 
process {S r , + W n }. For example, if C consists of all strictly 
causal finite-impulse response (FIR) filters of length p, the 
optimal filter C(Z) is the optimal predictor of S n from the 
samples {S„_i + W n -\,..., S n - P + W n - P }, which can be 
easily calculated in closed-form. 

The optimal sigma-delta filter design problem was studied 
by several authors, under various assumptions GDI, 0, ®, 
m-m. However, to the best of our knowledge, the simple 
expression from Theorem[2]for the optimal filter as the optimal 
predictor of S n from the past of {.S' r , + W n } is novel. The 
references most relevant to Theorem [2] are perhaps fl4ll 
and [i8], lfl5l . In fl4l . Spang and Schultheiss formulated an 
optimization problem for finding the best FIR filter with p 
coefficients in a sigma-delta modulator with a scalar quantizer, 
under a fixed overload probability. Their optimization problem 
can be solved numerically, but no closed form solution was 
given. In j8] and 1131 the design of an optimal unconstrained 
sigma-delta filter was studied, under the assumption of a fixed 
scalar quantizer which can only be scaled in order to control 
the overload probability. Equations that characterize the op¬ 
timal filter were derived. However, the obtained expressions 
usually yield filters with an infinite number of taps, and do 
not provide the solution to the constrained problem. It is also 
worth mentioning that for the case of a stationary Gaussian 
process { X n } with L = 1 (sampling at the Nyquist rate) 
and known PSD, the optimal infinite length filter under the 
assumption of high-resolution quantization is known to equal 
the optimal prediction filter of X n from its (clean) past Ifl2l . 
As already mentioned in the introduction, the high-resolution 
assumption never holds when L > 1 and therefore this result 
is inapplicable for over-sampled signals. 

Proof of Theorem^ By Proposition |T] if the test channel 




7 


from Figure Q] achieves MSE distortion D, we must have 


if-i/L li-CMlW 

By Theorem |T] the corresponding mutual information 
I(U^ A ;U^ A + N^ a ) is equal to the mutual information 
I(Un PCM 'i U® FCM + N° FCM ) in the DPCM test channel from 
Figure [2] with X% FCM = S n , N° FCM = W n and al ?CM = L-D. 
Thus, 


I(U* A ; U* A + N* a ) = C7™ + < PCM ) 


= 2 l0g 


1 + 


E (S n ~ C n * (S n + W n )) 
L-D 


(15) 


where we have used ([8}. ( | Kil l, and (ITTl i. to arrive at ( fl5l i. It 
follows that among all filters in C, the filter that minimizes (fl5T > 
is C E (Z), and that it attains (ITTb with equality. ■ 

It is interesting to note 0 that since { W n } is an i.i.d. 
process with variance L ■ D and C(Z) is strictly causal, the 
mutual information (fl5l) can also be written as 


i(u* a -,u* a + n* a ) 


= 2 log 


E (S n + W n — c n * (S n + W n )) 
L-D 


(16) 


Thus, the optimal predictor of S n from the past of { S n + W n } 
is identical to the optimal predictor of S n + W n from its past 
samples. When C(Z) is taken as the (unique) infinite order 
optimal one-step prediction filter of S n + W n from its past 
samples, the prediction error variance is the entropy power of 
the process {S n + W n } [16J, which equals 

2 J-* log (S s (u)+L-D)d,u = (£,.£)) ^1 q. 12L 

Moreover, the infinite order prediction error 

£T d = Sn + W n - c n * (Sn + Wn) 

is in this case a white process. This, together with (fT71 > implies 
that for the optimal unconstrained sigma-delta filter C(Z) we 
must have 



S e ^(oj) 4 |1 - C(u >)| 2 (L-D + S s (u)) 

( a ' 2 \ 1/L 

= i L ■ D ) (1 + -jj J , Vw e [-7T, 7r) (18) 

Combining ( IT6l >. (TlTb . and ( IT8l ) yields the following corollary. 

Corollary 1: Let {X^ A } be a Gaussian stationary process 
with variance a x whose PSD is zero for all oj ^ [— 7 t/L, 7 t/L\. 
If the test channel from Figure Q] attains MSE distortion D, 
then 

J(C/„ EA ; t/ n SA + TO > ^ log (l + ^) • (19) 


with equality if and only if C(Z) is a strictly causal filter 
satisfying 


|i-cMI 2 = 



and 


D 




( 1 + A) 


L ’ LI 


( 20 ) 


w e --r, -r 


7 T 7 T 1 

L’ L J’ 


L-D 


— (L — l)/L ' 


Remark 2: Note that the existence of a strictly causal filter 
C(Z) which satisfies ( l20l ) is guaranteed by Wiener’s spectral- 
factorization theory (ED due to the readily verified fact that 

2^/2 7r i°gl 1 - c 'MI 2d ^ _ 1 

The optimal filter induces a two-level frequency response for 
|1 — C(uj) | 2 . In fill 0stergaard and Zamir used sigma-delta 
modulation to attain the optimal multiple-description rate- 
distortion region. Interestingly, the optimal filter C(Z) in their 
scheme also induced a two-level response for |1 — C(ui) | 2 . 
We also note that the optimality of the unconstrained filter 
specified by (23 can be deduced as a special case of El 
Section IV]. 

Remark 3: Note that for the optimal unconstrained filter 
C(Z) specified by (l20l >. the pre- and post-filters from Figure [3] 
have no effect as long as the PSD of the input signal {X// A } 
is zero for all ui (/ [— tt/L, 7t/L]. However, filters with a finite 
number of taps will never incur a flat frequency response in 
the interval [—7 t/L, 7 t/L], and for such filters the systems from 
Figure Q] and Figure [3] will not be equivalent. 

Remark 4: The output of the test channel from Figure Q] 
(as well as that from Figure 0 is of the form X^ A = 
X^ A + E^ A , where E^ A has zero mean and variance D, 
and is statistically independent of X// A . This estimate can 
be further improved by applying scalar MMSE estimation for 
Xy‘ A from X// A . This boils down to producing the estimate 
1^ A = aX* A , where 

a'x 

a = —tt- 

<7 X +D 

Consequently, the obtained MSE distortion is reduced to 

n tcvvSA „ ySA\2 a 'x ' D 
D = ^(X n -aX n ) = 2 ■ 

a x ' u 

It is straightforward to verify 021 that with this improvement, 
the sigma-delta test channel from Figure Q] with C(Z) and 
cr|; A as specified in Corollary Q] attains 

r(t7“;t;“ + v“) = Ti og (i), 

which is the optimal rate-distortion function for a stationary 
Gaussian source { X)f A } with PSD as in ( 0 . It follows that the 
sigma-delta test channel from FigureQ]with C(Z) and o^ A as 
specified in Corollary 0 is minimax optimal for the class of all 
stationary Gaussian sources with variance rr‘ 2 x and PSD that 









equals zero for all u> ^ [—ir/L,Tr/L\, i.e., no other system 
can achieve MSE distortion D with a smaller compression 
rate, universally for all sources in this class. 

A. Extension to Frequency-Weighted Mean Squared Error 
Distortion 


• It follows that the DPCM test channel for the process 
{S n } under MSE distortion is equivalent to the sigma- 
delta test channel with input {X^ A } under FWMSE 
distortion, in the sense that in both channels if the attained 
distortion is -Dfwmse (under the appropriate distortion 
measure), then 


In many applications, higher values of distortion are ac¬ 
ceptable in certain frequency bands while smaller distortion 
is permitted in other bands. The MSE distortion measure is 
inadequate for such scenarios, and a commonly used distortion 
measure, that (partially) captures such perceptual effects, is the 
frequency-weighted mean squared error (FWMSE) criterion. 
Under this criterion, the distortion is measured as 

-Dfwmse — zfff J P(ut)SE{u)duj, (21) 

where P(ui) is a non-negative weight function, and Se{w) is 
the PSD of the error process E n = X^ A — X^ A . Note that 
for P(ui) = 1 ,Vw € [—7r,7r), the FWMSE criterion reduces 
to the MSE one. The next theorem shows that the constrained 
optimal sigma-delta filter under the FWMSE criterion is the 
optimal constrained prediction filter of a noisy process defined 
according to the weight function P(u>). 

Theorem 3: Let {2f^ A } be a Gaussian stationary process 
with variance a\ whose PSD is zero for all u ^ [—w/L, t t/L\, 
P(ui) a weighting function which forms a valid PSD, and C 
a family of strictly causal filters. Define the “virtual” process 
{S^} as a Gaussian stationary process with PSD 


FWMSE 


M = 


Ltj\P(w) for |w| < ttIL 


0 


for 7 t/L < |w| < 7T 


( 22 ) 


and the “virtual” process { W n } as a Gaussian i.i.d. ran¬ 
dom process statistically independent of { S n } with variance 
L - -Dfwmse- Dfwmse > 0. Let 


Dfwmse = E (5„ - C n * (5„ + W n )) 

C* D7WM J z ) = argminE (S n - c n * (S n + W n )) 2 . 

C(Z)GC 

If the filter C(Z) in the sigma-delta test channel from Figure Q] 
belongs to C and the FWMSE distortion w.r.t. P(w) attained 
by this test channel is Dfwmse. then 


1 / (J* Z 

rSA\ \ I i i Dfwmse 




J(C/^ A ; U^ A + N^) >-log 1+ - , , 

Z \ L • -/DFWMSE J 


with equality if C\Z) = C^JZ). 

Sketch of proof:: The proof is fairly similar to that of 
Theorem U Thus, for brevity, we omit the full proof and only 
highlight its main steps: 

• Repeat the derivation of Proposition [T] where now the 
MSE distortion is replaced by FWMSE distortion. Note 
that this has no effect on /(t/^ A ; U^ A + N^ A ). 

• Repeat the derivation of Proposition [2] where the PSD of 
the input process is (l22l >. rather than ©. Note that this 
changes J(t/° PCM ; f7° PCM + iV° PCM ), but has no effect 
on the attained distortion. 


/(U n SA ; t/„ SA + iV„ SA ) = /((/“; f/° PCM 


■N 


DPCM \ 


= 2 log 


1 + 


E(S n -c n *(S n + W n )f 
L ■ Dfwmse 


B. Sigma-Delta Modulation with an Interleaved Vector Quan¬ 
tizer 


The goal of this short subsection is to give the test channel 
from Figure [H an operational meaning, i.e., to show how 
the AWGN from the figure can be replaced with a lossy 
source code of rate R = I(U^ A ;U^ A + N^ A ) whose 
incurred quantization noise is distributed as N^ A . As already 
mentioned, the key idea is to use an interleaver EH3U, as 
we now recall. 

Assume that {X^ A }, the input process to the sigma- 
delta modulator, has a decaying memory, such that Xf A is 
essentially independent of all samples of sufficiently distant 
sampling times. In order to compress an A : -dimcnsional vector 

x — Lm > • • • > a at 

containing N consecutive samples of the process {2T^ A }, we 
first split it into I\ vectors 


= [*?£ 


1)M+1’ 


ySAi 

i -^kM J > 


k = 1, 


,K, 


where M = N/K. Now, we can apply K parallel sigma-delta 
modulators, one for each such vector, where the only coupling 
between the K parallel systems is through the quantization 
step, which is applied jointly on all of them, as depicted 
in Figure 0] By our assumption that { A aa } has decaying 
memory, if M is large enough the K inputs that enter 
the quantizer Q(-) = [Qi(-), ■ ■ ■ ,Qk(')] are i.i.d. random 
variables distributed as Uf A from Figure Q] For large enough 
K, standard rate-distortion arguments imply that there exists a 
vector quantizer with rate I(U^ A ; U^ A + N^ A ) that induces 
quantization noise distributed as N^ A . 


III. Sigma-Delta Modulation with a Scalar 
Uniform Quantizer 

The previous subsection showed how to replace the AWGN 
channel in Figure [j] with a vector quantizer whose rate is 
arbitrarily close to R = I{U^ A \U^ A + N^ A ) and whose 
induced quantization noise is distributed as N^ A . The inputs 
to the vector quantizer are vectors of i.i.d. Gaussian compo¬ 
nents. Thus, any “off-the-shelf” rate-distortion optimal vector 
quantizer for an i.i.d. Gaussian source can be used. The total 
sigma-delta compression system that is obtained is therefore 
simple in the sense that it only requires the vector quantizer 
to be good for quantizing an i.i.d. Gaussian source, which is 
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>EA 


-EA 

i K 


Fig. 4. K parallel sigma-delta modulators coupled by an K -dimensional quantizer Q(-'j- 


a standard task, rather than requiring it to be a good quantizer 
for a band-limited Gaussian source. 

However, the sigma-delta modulation architecture is mainly 
used for A/D and D/A conversion. In such applications, 
vector quantization is typically out of the question, and simple 
uniform scalar quantizers of finite support are used. For such 
quantizers, the quantization error is composed of two main 
factors (T): granular errors that correspond to the quantization 
error in the case where the input signal falls within the 
quantizer’s support, and overload errors that correspond to 
the case where the input signal falls outside the quantizer’s 
support. Due to the feedback loop, inherent to the sigma-delta 
modulator, errors of the latter kind, whose magnitude is not 
bounded, may have a disastrous effect as they jeopardize the 
system’s stability. In order to avoid such errors, the support of 
the quantizer has to be chosen appropriately. As the support 
of the quantizer determines its rate for a given quantization 
resolution, the overload probability can be controlled by 
increasing the quantization rate@ 

We shall show that, given that overload errors did not occur, 
the quantization noise can be modeled as an additive noise. 
Thus, the test channel from Figure |T] accurately predicts the 
total distortion incurred by a sigma-delta A/D (or D/A) in this 
case. Moreover, the overload probability is a doubly expo¬ 
nentially decreasing function of R — I(U^ A ; U^ A + N^ A ), 
where 2 R are the number of levels in the scalar quantizer. 
Thus, fixing the desired overload error probability as P a i, we 
may achieve the MSE distortion predicted by the test channel 
from Figure Q] (characterized in Proposition Q]) with a scalar 
quantizer whose rate is I(U^ A ] U„ A + N^ A ) + S(P 0 i), where 
S(Pol) = O (log log (prj) j- 

Let Qn a ‘i (■) be a uniform quantizer with quantization step 
sjYla 2 and 2 R quantization levels, such that the quantizer 
support is [— r/2, r/2), where T = 2 R s/l2a 2 , see Figure [5] 
Our goal is to analyze the distortion and overload probability 
attained by a sigma-delta modulator that uses a Qn a 2 (•) 
quantizer, as a function of R and crf^. 

2 As discussed in Section |I-B| one can try to limit the effect of overload 
errors by placing various constraints on C(Z). Here, we restrict attention to 
controlling the overload probability. 


Qr,c t 2 (x) 

3 


-4 


-2 


1 



+ 

2 


4 


-3 


Fig. 5. An illustration of Q R ^2 (-) for R = 2 and cr 2 = 1/3. 


Clearly, if we employ the scalar sigma-delta modulator on a 
long enough input sequence, an overload event will eventually 
occur. As discussed above, the effects of overload errors can 
be amplified due to the feedback loop, and in this case the 
average MSE may significantly grow. We therefore split the 
input sequence into finite blocks of length N , and initialize the 
memory of the filter C(Z) with zeros before the beginning of 
each new block. This makes sure that the effect of an overload 
error in the original system is restricted to the block where it 
occurs. 

The analysis is made much simpler by introducing a sub¬ 
tractive dither ED- Namely, let {Z n j be a sequence of 
i.i.d. random variables uniformly distributed over the interval 
[— y/l2a^ A /2, ^\2a\^l2}). In order to quantize U^ A , we 
add Z n to it before applying the quantizer, and subtract Z n 
afterwards, such that the obtained result is Qr.c^ a {U^ a + 
Z n ) — Z n . Adding and subtracting U^ A . we get U^ A + 
(QR,al A (Un A + Z n ) - {U„ A + Z n )j, and the quantization 
error is therefore 

Nn = Q R ,a | A (U* A + Z n ) - ([/„ SA + Z n ) (23) 

The main result in this section is the following. 

Theorem 4: Let D be the MSE distortion attained by the 
test channel in Figure |T] with a filter C(Z) of finite length, and 
I(U ^ A ; U^ A + N^ A ) the scalar mutual information between 
the input and output of the AWGN channel in the same figure. 
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Fig. 6. A sigma-delta modulator with a dithered scalar uniform quantizer. The input is assumed to be over-sampled at L times the Nyquist rate, and the 
dither sequence { Z n } is assumed to be an i.i.d. sequence of random variables uniformly distributed over the interval | \j 12 a 2 A /2, and 

statistically independent {A'^ a }. 


For any 0 < P 0 i < 1 the scalar sigma-delta modulator from 
Figure [6] applied on a sequence of N consecutive source sam¬ 
ples with quantization rate R = I(U^ A - U^ a +N^ a )+S(P 0 i) 
attains MSE distortion smaller than 
D(1 + o n {1)) 

1 -Pol ’ 


given that overload did not occur. In addition, the overload 
probability is smaller than P 0 i, where ojv(1) —> 0 as N 
increases, and 


S(ft<> = 5l°g(-flnf|). (24) 


Proof: Let 2 Z (aO be the operation of rounding x 

to the nearest point in the (infinite) lattice y/l2cr 2 Z. It is easy 
to verify that for any x £ [—T/2, T/2) we have 


Qr,c 2 0 ) 



Vl2cr 2 

2 


(25) 


Applying (l23l > therefore yields that if overload did not occur 
in the ?rth sample, i.e., if | U^ A + Z n \ < T/ 2, we have 


N n = Q 

-(u. 


y/12^. 

EA 


u„ A + Z n + 


3o SA 


3<t ea 


(26) 


Dealing with the overload event of the quantizer directly is 
rather involved. Instead, as done in HE), we first consider a 
reference system with an infinite-support quantizer (R = oo) 
and analyze its performance. If the magnitude of the input 
to the infinite-support quantizer never exceeds F/2 within 
the processed block, then clearly the reference system is 
completely equivalent to the original system within this block. 
Thus, it suffices to find the average distortion of the reference 
system and the probability that the input to its quantizer 
exceeds F/2 within a block. In what follows we will therefore 
assume that the quantization noise is given by (f26l) regardless 
of whether or not \U^ A + Z n \ < T/2, and account for the 
overload probability later. 

Assuming that the dither sequence {Z n j is drawn sta¬ 
tistically independent of the process {X^ A }, the Crypto 
Lemma, see, e.g. G 3 Lemma 4.1.1], implies that {A^ n } is an 
i.i.d. sequence of random variables uniformly distributed over 


the interval [—y / 12cr| A /2, y / 12cr| A /2), statistically indepen¬ 
dent of {A’ r ^ A }. Note that N n has zero mean and variance 
erf; a- Following this reasoning, the reference sigma-delta 
data converter depicted in Figure [6] (with an infinite-support 
quantizer) is equivalent to the test channel from Figure Q] 
with N^ A ~ Uniform ^[— y/l2crf A /2, -y/l2crf A /2)^ instead 
of N^ a ~ Af(0, erf A ). Thus, the average MSE distortion 
attained by the reference scalar sigma-delta modulator from 
Figure [6] is as given in Proposition Q] up to a multiplicative 
factor of l+oj\r(l) that accounts for edge effects. These effects 
are the by-product of the operation of nulling the filter memory 
at the beginning of each new block, which incurs temporal 
non-stationarities. In particular, if the filter C{Z) has L taps, 
then only after L samples within the block the statistics of 
the process {U^ A } will converge to its stationary distribution. 
However, if the block length is sufficiently large w.r.t. the filter 
length and the inverse of the MSE distortion, the influence of 
these effects vanishes. 

Next, we turn to analyze the probability that an overload 
error occurs within a block of length N, as a function of R 
and I(U^ A ; U^ a +N^ a ). Since this event is equivalent to the 
event that at the reference system some input to the quantizer 
exceeds T/2 in magnitude within the block, it suffices to upper 
bound the probability of the latter event. 

Assume the reference scalar sigma-delta modulator from 
Figure [6] is applied to a vector x SA = [Xf :A ...., X^, A ] of 
N consecutive samples of the process { Xf A }, where the 
memory of the filter C(Z) is initialized with zeros. Define 
the event OL^ = {|(7| ]A + Nj^ A \ > T/2} and the event 
OL = U A OLfc. By the union bound, we have 


N 

P ol 4 Pr(OL) < Y , Pr (° L fc) ■ (27) 

k =1 


The random variable U^ A + Nf A = X^ A + (Sk — Ck) * 
N AA is a linear combination of a Gaussian random variable 
X AA and statistically independent uniform random variables 
{ N AA }. In lfl9l Lemma 4] the probability that a random 
variable of this type exceeds a certain threshold was bounded 
in terms of its variance. Applying this bound to U^ A + Nj^ A 
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yields 
Pr | \Ul 


+ Nj? A \ > r/2 I < 2 exp < — 


= 2 exp < — 


8E m A + N^ A y 


124a2 M 


8(E (C/ fc SA ) 2 


■e(^ ea ) 2 ) 



where in the last equality we have used the definition of T 
and the fact that Uj/ A and Nj// A are statistically independent. 
Equivalently, we may write 


probability smaller than 2iV exp — ^2 2|S j-. Thus, Proposi¬ 
tion Q] characterizes the rate-distortion tradeoff achieved by 
the scalar sigma-delta system up to the aforementioned factor 
and a constant rate penalty S(P 0 i), that depends on the 
target overload error probability. To be more precise, for any 
0 < P 0 i < 1, taking the rate penalty as in < 124b guarantees that 
the overload error probability is smaller than P Q i. ■ 

Acknowledgements 


Pr OLfc I < 2 exp < — 


12a| A 2 2 « 




= 2 exp • 

= 2 exp■ 


2 


A 2 2 ( r - i ( u ^’ u ^ +n ^)) J , (28) 


where we have used (0 in the last equality. Substituting < 12 8 b 
into (Gill gives 



Note that E(Uj/ A ) 2 = a\ + a^ A Yfn =l c fc * s monotonically 
nondecreasing in k and is given by © for values of k that 
are greater than the length of the filter Ck . We can therefore 
further bound (l29t as 

Poi < 2N exp | —|2 2 ( fl ” / ( c/ " A;C/ " A+JV " A ))| , (30) 

where I (U// A ; U// A + N// A ) is as given in Proposition Q] To 
summarize, we have shown that the reference system achieves 
the same MSE distortion D as characterized by Proposition Q] 
up to a 1 + oat(1) multiplicative term, and that the probability 
that one of the quantizer input samples exceeds T/2 in 
magnitude within a block of length N, is bounded by <f30b . 
For our original system whose quantizer has finite support of 
[—r/2, r/2), this means that the overload probability is also 
upper bounded by the RHS of <l30l >. Moreover, the average 
distortion it achieves if overload did not occur is the same as 
that of the reference system conditioned on the event that OL 
did not occur. Denote this conditioned expected distortion by 
Dqp and the expected distortion conditioned on the event that 
OL did occur by ZJql- For the reference system, we have 


0(1 + oiv(l)) = Pr(OL)0 5E + Pr(OL)D OL > Pi/OLjO^, 


and therefore 

„ ^ 0(1 + ojv(l)) 

OL - 1 -Pol ' 

This shows that the scalar sigma-delta system from Fig- 
ureO whose quantizer has limited support [—T/2, T/2), with 
R = I{U// A -, U// a +N// a )+5(P 0 i) achieves the same average 
MSE distortion as the test channel from Figure Q] up to a 
multiplicative factor of (1 + ojv(1))/(1 — Pol), with block error 
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